Soundspotter / remix-TV: Fast Approximate Matching for audio and video Performance

SoundSpotter is an open source software system for real-time matching of an audio input stream to a database of continuous audio or video. Among its novel features are real-time control over audio segmentation, feature selection and match radius. The system uses audio input to control selection of output from a database using similarity-based matching. The low latency methods employed create a feedback loop between the performer and the database, thus it is a type of electronic musical instrument. We employ exact nearest neighbor searching on variable-length sequences of audio features for matching. The feature space is controlled by range selection over the vector dimensions determining which are left out of similarity calculations. The current implementation is capable of real-time matching in tens of hours of audio or video on current laptop hardware. Finally, we describe SoundSpotter’s video editing utility in an application called REMIX-TV.